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SUBJECT:  Report  on  Implementation  of  Performance-Based  Logistics  for  the  Joint 
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We  are  providing  this  report  for  review  and  comment.  We  considered 
management  comments  on  a  draft  of  this  report  in  preparing  the  final  report, 

DoD  Directive  7650,3  requires  that  all  recommendations  be  resolved  promptly. 
The  comments  provided  by  the  Military  Deputy,  Office  of  the  Assistant  Secretary  of  the 
Air  Force  (Acquisition)  on  behalf  of  the  Assistant  Secretary  of  the  Air  Force 
(Acquisition)  were  responsive  on  all  but  one  recommendation.  We  request  that  the 
Assistant  Secretary  of  the  Air  Force  (Acquisition)  provide  comments  clarifying  his 
position  on  Recommendation  2,e.  by  August  25,  2006. 

If  possible,  please  send  management  comments  in  electronic  format  (Adobe 
Acrobat  file  only)  to  AudRLS@dodig.mil.  Copies  of  the  management  comments  must 
contain  the  actual  signature  of  the  authorizing  official.  We  cannot  accept  the  /  Signed  / 
symbol  in  place  of  the  actual  signature.  If  you  arrange  to  send  classified  comments 
electronically,  they  must  be  sent  over  the  SECRET  Internet  Protocol  Router  Network 
(SIPERNET). 

We  appreciate  the  courtesies  extended  to  the  staff.  Questions  should  be  directed 
to  Mr.  Robert  F.  Prinzbach  at  (703)  604-8907  (DSN  664-8907)  or  Mr.  Keith  A.  Yancey 
at  (703)  604-8774  (DSN  664-8774).  See  Appendix  C  for  the  report  distribution.  The 
team  members  are  listed  inside  the  back  cover. 
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Implementation  of  Performance-Based  Logistics  for  the 
Joint  Surveillance  Target  Attack  Radar  System 


Executive  Summary 


Who  Should  Read  This  Report  and  Why?  DoD  personnel  and  Government 
contractors  responsible  for  implementing  perfonnance-based  logistics  (PBL)  should  read 
this  report.  The  report  discusses  implementation  of  PBL  for  the  Joint  Surveillance  Target 
Attack  Radar  System  (Joint  STARS). 

Background.  PBL  is  the  DoD-preferred  approach  for  implementing  product  support. 

The  approach  is  a  strategy  for  weapon  system  life-cycle  sustainment  that  links  product 
support  to  weapon  system  perfonnance.  The  goal  of  PBL  is  to  optimize  total  system 
availability  while  minimizing  cost  and  the  logistics  footprint.  Logistics  footprint  is  the 
amount  of  support  required  to  deploy,  sustain,  and  move  a  weapon  system.  PBL  places 
accountability  for  product  support  on  the  program  manager.  The  program  manager  may 
use  organic  (DoD)  support,  commercial  sector  support,  or  a  partnership  between  organic 
and  commercial  providers.  The  PBL  may  involve  arrangements  between  public  and 
private  sectors  or  between  two  public  organizations.  The  Assistant  Deputy  Under 
Secretary  of  Defense  (Logistics  Plans  and  Programs)  oversees  implementation  of  PBL. 

The  Defense  Planning  Guidance  for  FY  2003  through  FY  2007  required  that  each 
Military  Department  submit  a  plan  that  would  identify  its  implementation  schedule  for 
applying  PBL  to  any  new  weapon  system  as  well  as  any  Acquisition  Category  I  and  II 
fielded  system.  On  June  25,  2004,  the  Air  Force  identified  that  it  implemented  PBL  for 
Joint  STARS. 

The  Joint  STARS  aircraft,  which  is  an  Acquisition  Category  I  program,  is  a  joint 
development  project  of  the  Air  Force  and  the  Army.  The  aircraft  provides  an  airborne, 
stand-off  range,  surveillance  and  target  acquisition  radar  and  coimnand  and  control 
center.  Joint  STARS  conducts  ground  surveillance  and  helps  commanders  understand  an 
enemy  situation  as  well  as  supports  attack  operations  and  targeting.  Joint  STARS 
detects,  locates,  classifies,  tracks,  and  targets  hostile  ground  movements,  communicating 
real-time  information  through  secure  data  links  between  Air  Force  and  Army  coimnand 
posts.  The  Joint  STARS  was  first  deployed  in  Operation  Desert  Stonn  in  1991  and  began 
full  rate  production  of  14  aircraft  in  September  1996.  As  of  December  1,  2005,  17 
aircraft  are  in  the  Air  Force  inventory. 

Results.  The  Joint  STARS  System  Program  Manager  did  not  fully  implement  PBL 
initiatives  for  the  Joint  STARS  weapon  system.  As  a  result,  the  System  Program 
Manager  cannot  support  that  the  Joint  STARS  weapon  system  is  achieving  the  desired 
outcomes  of  PBL,  such  as  reducing  life-cycle  costs  and  increasing  system  availability. 

We  recommend  that  the  Assistant  Secretary  of  the  Air  Force  (Acquisition)  distribute 


DoD  policy  memorandums  and  guides  for  implementing  performance-based  logistics. 

We  also  recommend  that  the  System  Program  Manager  develop  perfonnance-based 
agreements  with  the  warfighter,  develop  a  strategy  that  implements  performance 
requirements,  and  analyze  Joint  STARS  cost  and  resource  data  and  develop  baselines. 
Further,  we  recommend  that  the  System  Program  Manager  develop  a  business  case 
analysis  based  on  warfighter  requirements  and  baselines,  develop  performance  measures 
that  promote  the  goal  of  increased  system  readiness  and  reductions  in  life-cycle  costs  and 
logistics  footprints  for  Joint  STARS,  and  analyze  transitioning  the  total  system  support 
responsibility  contract  to  a  firm-fixed  price  contract.  (See  the  Finding  section  for  the 
detailed  recommendations.) 

Management  Comments  and  Audit  Response.  The  Military  Deputy,  Office  of  the 
Assistant  Secretary  of  the  Air  Force  (Acquisition)  provided  comments  on  behalf  of  the 
Assistant  Secretary  of  the  Air  Force  (Acquisition).  The  Military  Deputy  partially 
concurred  with  our  draft  finding  that  the  System  Program  Manager  did  not  fully 
implement  the  PBL  initiative  for  Joint  STARS.  The  Military  Deputy  stated  that  the  Joint 
STARS  System  Program  Manager  did  not  retroactively  apply  PBL  principles  as  a  result 
of  the  total  system  support  contract,  which  was  a  model  for  sustainment  approach  before 
the  PBL  guidance. 

The  Military  Deputy  concurred  with  our  recommendation  to  distribute  DoD  policy 
memorandums  and  guides  to  each  Product  Center  for  implementing  PBL  and  with  our 
recommendation  to  analyze  Joint  STARS  cost  and  resource  data  as  well  as  develop 
baselines  for  developing  a  business  case  analysis.  The  Military  Deputy  also  agreed  with 
our  recommendation  to  analyze  the  impact  of  transitioning  the  total  system  support 
contract  from  a  cost-plus  contract  to  a  firm-fixed  priced  contract.  No  further  comments 
are  required. 

The  Military  Deputy  also  partially  concurred  with  our  recommendation  to  develop 
performance-based  agreements  and  a  strategy  that  implements  performance  requirements 
from  those  agreements  into  the  Joint  STARS  total  system  support  responsibility  contract. 
He  stated  that  improvements  will  be  made  to  the  scope,  objective,  and  perfonnance  of  the 
performance-based  agreement.  The  Military  Deputy  partially  concurred  with  our 
recommendation  to  develop  a  business  case  analysis  using  baselines,  stating  a  forward- 
looking  business  case  analysis  will  be  conducted  in  accordance  with  policy  guidance  and 
assess  alternative  solutions  that  will  meet  the  objectives  of  the  warfighter.  The  Air  Force 
comments  are  responsive,  and  no  further  comments  are  required. 

The  Military  Deputy  partially  concurred  with  our  recommendation  to  develop 
performance  measures  for  the  Joint  STARS  contract  that  will  promote  the  goal  of 
increasing  system  readiness,  reducing  life-cycle  costs,  and  reducing  the  logistics 
footprint.  However,  the  Military  Deputy  stated  that  those  goals  were  already  in  the 
memorandum  of  agreement.  The  Air  Force  comments  were  partially  responsive.  The 
memorandum  of  agreement  was  not  clear  that  it  contained  established  goals  of  increasing 
system  readiness,  reductions  in  total  life-cycle  costs,  and  reductions  in  the  logistics 
footprints.  We  request  that  the  Assistant  Secretary  of  the  Air  Force  (Acquisition)  clarify 
his  response  to  develop  new  performance  measures  in  meeting  those  goals  and  provide 
additional  comments  to  the  final  report  by  August  25,  2006.  See  the  Finding  section  of 
the  report  for  a  discussion  of  management  comments  and  the  Management  Comments 
section  for  the  complete  text  of  comments. 
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Background 


Performance-Based  Logistics.  Performance-based  logistics  (PBL)  is  the 
DoD-preferred  approach  for  implementing  product  support.  The  approach  is  a 
strategy  for  weapon  system  life-cycle  sustainment  that  links  product  support  to 
weapon  system  perfonnance.  The  goal  of  PBL  is  to  optimize  total  system 
availability  while  minimizing  costs  and  the  logistics  footprint.  Logistics  footprint 
is  the  amount  of  support  required  to  deploy,  sustain,  and  move  a  weapon  system. 
PBL  places  accountability  for  product  support  with  the  program  manager.  The 
program  manager  has  several  options  available  for  obtaining  product  support. 
Those  options  include  areas  such  as  organic  (DoD)  support,  commercial  support, 
or  a  partnership  between  organic  and  commercial  providers. 

Office  of  the  Secretary  of  Defense  Implementation  of  PBL.  In  September 
2001,  the  Quadrennial  Defense  Review1  mandated  implementation  of  PBL  for 
DoD  acquisitions.  The  Under  Secretary  of  Defense  for  Acquisition,  Technology, 
and  Logistics  (USD[AT&L])  assigned  the  Assistant  Deputy  Under  Secretary  of 
Defense  (Logistics  Plans  and  Programs)  with  the  responsibility  of  overseeing  the 
DoD  implementation  of  PBL. 

Key  Guidance  for  Implementing  PBL.  Since  the  2001  Quadrennial  Defense 
Review  mandated  implementation  of  PBL,  the  USD(AT&L)  and  the  Air  Force 
have  issued  policy  memorandums,  support  guides,  and  instructions  designed  to 
facilitate  implementing  PBL  for  DoD  acquisitions,  both  new  and  fielded. 
Guidance  includes  the  following: 

•  USD(AT&L)  memorandum,  dated  February  13,  2002,  states  that  the 
FY  2003-07  Defense  Planning  Guidance  requires  that  each  Military 
Department  submit  a  plan  identifying  its  implementation  schedule  for 
applying  PBL  to  all  Acquisition  Category  I  and  II2  new  and  fielded 
weapon  systems.  The  memorandum  provides  guidelines  for  preparing 
PBL  implementation  schedules  as  well  as  information  that  further 
guidance  for  developing  PBL  strategies  and  implementation  is  contained 
in  the  November  2001  document,  “Product  Support  for  the  21st  Century: 

A  Program  Manager’s  Guide  to  Buying  Performance.”  The  guide 
requires  that  Program  Managers  implement  PBL  on  new  systems  and  on 
Acquisition  Category  I  and  II  fielded  systems  on  the  basis  of  a  sound 
business  case  analysis  that  looks  at  alternative  support  strategies. 

•  DoD  Directive  5000. 1,  “The  Defense  Acquisition  System,”  May  12,  2003, 
establishes  policies  for  acquisition  programs  in  DoD.  The  Directive  states 


1  The  Quadrennial  Defense  Review  is  the  strategic  planning  document  for  DoD  to  transform  its  defense 
planning  to  a  model  capable  of  dealing  with  threats  in  the  21st  century. 

2  The  acquisition  category  determines  an  acquisition  program’s  level  of  review,  decision  authority,  and 
applicable  procedures.  Acquisition  Category  I  programs  are  acquisition  programs  with  an  estimated  total 
expenditure  for  research,  development,  test,  and  evaluation  of  more  than  $365  million  or  procurement  of 
more  than  $2.19  billion.  Acquisition  Category  II  programs  have  an  estimated  total  expenditure  for 
research,  development,  test,  and  evaluation  of  $365  million  or  less  but  more  than  $140  million  or 
procurement  of  $2.19  billion  or  less  but  more  than  $660  million. 
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that  Program  Managers  must  develop  and  implement  PBL  strategies  that 
optimize  total  system  availability  while  minimizing  cost  and  the  logistics 
footprint. 

•  In  November  2004,  the  “Performance  Based  Logistics:  A  Program 
Manager’s  Product  Support  Guide”  superseded  the  Program  Manager’s 
Guide  to  Buying  Performance.  The  Product  Support  Guide  is  designed  to 
help  managers  develop  product  support  strategies  for  new  programs  and 
help  reengineer  product  support  for  existing  fielded  systems. 

•  Air  Force  Instruction  63-107,  “Integrated  Product  Support  Planning  and 
Assessment,”  November  10,  2004,  contains  the  PBL  guidance  for  the  Air 
Force.  The  Instruction  provides  policy  for  integrated  product  support 
planning  and  assessment  for  implementation  of  PBL.  The  Instruction 
states  that  the  program  manager  must  ensure  that  logistics,  as  a  process  for 
weapon  system  support,  serves  two  key  objectives:  (1)  system  readiness  is 
increased  through  integrated  logistics  and  (2)  resources  required  to  fulfill 
support  requirements  are  minimized. 

Implementing  PBL  Initiatives  for  the  Joint  Surveillance  Target  Attack 
Radar  System.  The  Joint  Surveillance  Target  Attack  Radar  System  (Joint 
STARS)  aircraft  is  an  Acquisition  Category  I  program.  The  system  is  a  joint 
development  project  of  the  Air  Force  and  the  Army  that  provides  airborne,  stand¬ 
off  range,  surveillance  and  target  acquisition  radar  and  command  and  control.  It 
can  conduct  ground  surveillance  to  help  commanders  understand  the  enemy 
situation  and  support  attack  operations  and  targeting.  As  of  December  1,  2005, 
the  Air  Force  had  17  operational  Joint  STARS  aircraft  in  its  inventory.  The 
1 16th  Air  Control  Wing  (warfighter),  Robins  Air  Force  Base,  Georgia,  is  the 
primary  user  of  the  aircraft.  The  program  management  office  for  Joint  STARS  is 
the  System  Program  Manager  (SPM),  located  at  the  Electronic  Systems  Center, 
Hanscom  Air  Force  Base,  Massachusetts.  The  System  Support  Manager  (SSM), 
Warner  Robins  Air  Logistics  Center  at  Robins  Air  Force  Base,  manages  the  Joint 
STARS  sustainment  responsibilities  for  the  SPM. 

In  response  to  the  FY  2003-07  Defense  Planning  Guidance  requirement  to  submit 
a  plan  identifying  a  schedule  for  applying  PBL  to  all  new  weapon  systems  and  all 
Acquisition  Category  I  and  II  fielded  systems,  the  Air  Force  stated  in  June  2004 
that  the  Joint  STARS  program  implemented  PBL  initiatives.  The  Air  Force  made 
its  decision  based  on  the  September  2000  award  of  a  contract  to  Northrop 
Grumman  Corporation  (NGC)  for  total  system  support  responsibility  (TSSR). 

The  Air  Force  supported  its  decision  on  several  perfonnance  incentives  in  the 
TSSR  contract.  Under  the  TSSR  contract,  NGC  is  the  weapon  system  integrator 
providing  life-cycle  sustainment  support.  The  goals  of  the  TSSR  contract  are  to 
increase  weapon  system  availability  and  decrease  total  ownership  costs.  The 
TSSR  contract  has  a  6-year  base  period  with  a  potential  for  NGC  earning  as  many 
as  16  additional  years  for  a  maximum  value  of  $7  billion  over  the  life  of  the 
contract. 
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Objectives 


Our  overall  audit  objective  was  to  evaluate  the  effectiveness  of  the  Air  Force 
implementation  of  PBL  for  the  Joint  STARS.  Specifically,  we  assessed  the 
adequacy  of  the  business  case  analysis  (BCA)  and  determined  whether 
implementation  of  PBL  resulted  in  improvement  of  system  availability,  readiness, 
and  cost.  We  also  reviewed  the  management  control  program  as  it  related  to  the 
overall  objective.  See  Appendix  A  for  a  discussion  of  the  scope  and 
methodology.  See  Appendix  B  for  prior  coverage  related  to  the  objectives. 


Management  Control  Program  Review 


DoD  Directive  5010.38,  “Management  Control  Program,”  August  26,  1996,  and 
DoD  Instruction  5010.40,  “Management  Control  Program  Procedures,”  August 
28,  1996,  require  DoD  organizations  to  implement  a  comprehensive  system  of 
management  controls  that  provides  reasonable  assurance  that  programs  are 
operating  as  intended  and  to  evaluate  the  adequacy  of  the  controls. 

Scope  of  the  Review  of  the  Management  Control  Program.  We  reviewed  the 
adequacy  of  Air  Force  implementation  of  PBL  through  our  audit  of  the  Joint 
STARS  SPM  and  SSM  processes  and  procedures  for  implementing  PBL  for  the 
Joint  STARS  weapon  system.  We  also  reviewed  management’s  self-evaluation  of 
those  processes  and  procedures. 

Adequacy  of  Management  Controls.  The  Air  Force  SPM  and  SSM  did  not 
know  the  processes  and  procedures  USD(AT&L)  issued  for  PBL.  The  processes 
and  procedures  the  SPM  and  the  SSM  used  were  insufficient  to  ensure  full 
implementation  of  PBL  for  Joint  STARS.  USD(AT&L)  issued  PBL  policy 
memorandums  and  guidance  to  the  Air  Force.  Because  the  SPM  and  the  SSM 
stated  they  did  not  receive  USD(AT&L)  PBL  documentation  from  the  Air  Force 
and  were  unaware  of  its  requirements,  we  provided  them  copies  of  those 
documents.  We  do  not  consider  this  weakness  in  issuing  policies  and  procedures 
a  material  weakness  as  DoD  Instruction  5010.40  defines.  We  discussed  the 
weakness  with  personnel  from  USD(AT&L)  and  Air  Force,  who  stated  that 
reiteration  of  the  policies  and  procedures  for  implementing  PBL  were  needed. 

The  recommendation,  if  implemented,  will  correct  the  management  control 
weakness  cited  (see  Recommendation  1).  A  copy  of  the  report  will  be  provided  to 
the  senior  official  responsible  for  management  controls  in  the  Air  Force. 

Adequacy  of  Management’s  Self-Evaluation.  Neither  the  SPM  nor  SSM 
identified  initiatives  of  PBL  to  include  the  development  of  a  BCA,  performance 
baselines,  performance  outcomes,  and  performance-based  agreements  with  the 
warfighter,  as  an  assessable  unit.  Therefore,  the  program  office  did  not  identify 
or  report  the  control  weakness. 
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Implementation  of  Performance-Based 
Logistics  on  the  Joint  STARS 
Weapon  System 

The  Joint  STARS  SPM  did  not  fully  implement  PBL  initiatives  for  the 
Joint  STARS  weapon  system.  The  lack  of  full  implementation  occurred 
because  the  SPM  did  not  reassess  the  Joint  STARS  support  strategy  and 
incorporate  USD(AT&L)  guidance  for  implementing  PBL.  As  a  result, 
the  SPM  cannot  support  that  the  Joint  STARS  weapon  system  is  achieving 
the  desired  outcomes  of  PBL,  such  as  reducing  life-cycle  costs  and 
increasing  system  availability. 


Applying  PBL  Initiatives  to  DoD  Weapon  Systems 


“Performance  Based  Logistics:  A  Program  Manager’s  Product  Support  Guide” 
(Product  Support  Guide),  November  2004,  states  that  PBL  uses  a  performance- 
based  acquisition  strategy  that  is  developed,  refined,  and  implemented  during  the 
systems  acquisition  process  for  new  programs.  The  Product  Support  Guide 
further  states  that  the  strategy  is  the  result  of  an  assessment  of  performance 
and  support  alternatives  for  fielded  systems.  The  Product  Support  Guide 
describes  the  essence  of  PBL  as  buying  performance  outcomes,  not  the  individual 
parts  and  repair  actions.  Such  a  major  shift  from  the  traditional  approach  to 
product  support  emphasizes  what  level  of  support  program  managers  buy,  not 
who  they  buy  from.  In  a  memorandum  issued  by  the  Acting  USD(AT&L), 
“Performance  Based  Logistics:  Purchasing  Using  Perfonnance  Based  Criteria,” 
August  16,  2004,  it  stated  that  application  of  the  PBL  concept  may  take  place  at 
the  system,  subsystem,  or  major  assembly  level  depending  on  program  unique 
circumstances. 

The  Product  Support  Guide  presents  a  12-step  methodology  for  implementing 
PBL  that  is  applicable  to  new,  modified,  or  fielded  systems.  Those  12  steps  are: 

1 .  Integrate  requirements  and  support 

2.  Form  the  PBL  team 

3.  Baseline  the  system 

4.  Develop  performance  outcomes 

5.  Select  the  product  support  integrator 

6.  Develop  a  workload  allocation  strategy 

7.  Develop  a  supply  chain  management  strategy 

8.  Perform  a  PBL  business  case  analysis 

9.  Establish  performance  based  agreements 

10.  Award  contracts 

1 1 .  Employ  financial  enablers 

12.  Implement  and  assess 
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According  to  the  Product  Support  Guide,  the  PBL  implementation  process  is  not 
intended  as  rigid.  In  fact,  the  Product  Support  Guide  states  that  the  order  in 
which  steps  are  taken  can  be  flexible  and  allows  for  omission  of  some  steps.  Our 
review  of  PBL  implementation  for  the  Joint  STARS  weapon  system  focused  on 
four  of  the  key  steps  to  implementing  PBL:  establishing  perfonnance-based 
agreements  with  the  warfighter;  baselining  the  system;  developing  a  BCA;  and 
developing  performance  outcomes. 


Establishing  Performance-Based  Agreements  with 
the  Warfighter 


The  Joint  STARS  sustainment  strategy  lacks  support  of  fully  developed  PBAs 
with  the  warfighter. 

DoD  Instruction  5000.2,  “Operation  of  the  Defense  Acquisition  System,” 

May  12,  2003,  states  that,  “the  program  manager  shall  work  with  the  users  to 
document  perfonnance  and  support  requirements  in  perfonnance  agreements 
specifying  objective  outcomes,  measures,  resource  commitments  and  stakeholder 
responsibilities.” 

In  addition  to  the  Instruction,  the  Product  Support  Guide  states  that  a  PBA  creates 
a  clear  understanding  of  the  outcomes  and  the  commitments  required  for 
achieving  those  outcomes  among  stakeholder  parties.  Typically,  a  PBA  identifies 
the  warfighter’s  expected  perfonnance  goals  and  objectives,  such  as  availability 
and  cost,  and  establishes  a  target  price  based  on  the  desired  level  of  performance. 
Documentation  of  a  completed,  approved,  and  funded  PBA  with  a  targeted  price 
based  on  the  desired  outcome  is  a  critical  step  in  any  PBL  implementation. 

The  Joint  STARS  SSM  identified  two  memorandums  of  agreement  with  the 
warfighter  as  PBAs.  Those  memorandums,  one  dated  June  11,  1998,  and  the 
other  dated  September  15,  2004,  did  not  incorporate  the  warfighter’s  expected 
performance  goals,  objectives,  and  target  price  based  on  the  desired  level  of 
performance.  For  example,  an  Air  National  Guard  official  stated  that  having  only 
three  aircraft  in  scheduled  depot  maintenance  at  any  given  time  is  a  goal  of  the 
warfighter.  However,  this  goal  was  not  documented  in  a  PBA  or  an  updated 
memorandum  of  agreement.  In  addition,  the  goal  was  not  included  as  a 
performance  measure  in  the  TSSR  contract.  Our  review  of  the  records  for  the 
Joint  STARS  scheduled  depot  maintenance  showed  the  average  number  of 
aircraft  in  scheduled  maintenance  per  month  for  FY  2004  was  5.4  aircraft.  Had 
the  warfighter  goal  of  three  aircraft  in  scheduled  depot  maintenance  been  met  in 
FY  2004,  at  least  two  additional  aircraft  would  have  been  available  to  the 
warfighter.  Developing  PBAs  or  updating  the  existing  memorandums  of 
agreement  based  on  current  warfighter  requirements  will  help  ensure  the 
attainment  of  warfighter  goals  through  the  TSSR  contract. 
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Baselining  Costs  and  Performance 


The  Joint  STARS  SPM  did  not  develop  cost  and  performance  baselines  for  the 
Joint  STARS  weapon  system  before  the  August  2000  approval  by  Air  Force 
acquisition  officials.  In  addition,  when  the  Product  Support  Guide  was  issued,  the 
Joint  STARS  SPM  did  not  reassess  perfonnance  baselines. 

The  Product  Support  Guide  states  that  to  develop  an  effective  support  strategy,  a 
program  manager  should  identify  differences  between  existing  and  desired 
performance  requirements.  Therefore,  the  program  manager  identifies  and 
documents  the  current  performance  and  cost  baselines.  For  existing  systems, 
baseline  assessments  form  the  basis  for  the  BCA  of  PBL  approaches  being 
considered.  Detennination  of  the  sustainment  and  readiness  performance  history 
and  associated  operations  and  support  cost  is  essential.  Therefore,  actual  data, 
when  available,  should  be  used  for  fielded  systems. 

The  Defense  Acquisition  University  Web  site  contains  a  Toolkit  to  assist  a 
program  manager  when  creating  and  implementing  PBL.  One  section  of  this 
Toolkit  describes  baselines  as  the  starting  point  for  measuring  progress  in  the 
quality  or  quantity  of  work  or  perfonnance  related  to  either  a  product  or  service. 

The  section  also  states  that  baselines  are  not  necessarily  set  only  at  the 
beginning  of  a  program  but  should  be  reset  as  the  program  evolves  so  the 
program  manager  can  document  where  they  are  at  various  milestones  over 
the  life  cycle. 

The  Joint  STARS  program  office  began  developing  the  sustainment  strategy  in 
1998.  The  original  strategy  called  for  a  consolidation  of  up  to  1 1  contracts  into 
2  major  contracts — one  sole-source  and  one  competitive.  The  strategy  was 
revised  and  the  new  strategy  called  for  NGC  being  in  charge  of  sustainment, 
which  led  to  the  integration  of  sole-source,  competitive,  and  depot  efforts. 

Several  acquisition  and  weapon  system  personnel  questioned  the  revised  strategy 
because,  according  to  acquisition  personnel,  officials  needed  a  full  understanding 
of  the  proposed  costs  and  perfonnance  compared  with  the  current  baseline. 
Although  questions  about  the  baselines  for  Joint  STARS  were  asked,  acquisition 
officials  detennined  that  baselines  were  not  necessary  and  none  were 
documented.  As  of  November  29,  2005,  the  Joint  STARS  SPM  has  not 
reassessed  the  Joint  STARS  sustainment  strategy  originally  approved  in  August 
2000.  Now  that  production  of  Joint  STARS  aircraft  is  complete  and  more  than  5 
years  of  cost  and  perfonnance  data  are  available,  an  accurate  baseline  should  be 
established  and  used  in  the  development  of  a  BCA  and  measurement  of  the 
sustainment  strategy’s  performance. 


Developing  a  Business  Case  Analysis 


The  Joint  STARS  SPM  did  not  prepare  a  BCA  to  provide  analysis  of  potential 
alternative  support  strategies. 
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The  Product  Support  Guide  states  a  PBL  BCA  is  an  expanded  cost  and  benefit 
analysis  created  to  help  determine  a  best-value  solution  for  product  support.  The 
warfighter’s  stated  perfonnance  requirements,  which  are  documented  in  a  PBA, 
form  the  basis  for  a  BCA.  Development  of  a  PBL  BCA  should  determine  at  a 
minimum:  (1)  the  relative  cost  versus  benefits  of  different  support  strategies; 

(2)  methods  and  rationale  used  for  quantifying  benefits  and  costs;  (3)  impact  and 
value  of  performance,  cost,  schedule,  and  sustainment  trade-offs;  (4)  data  required 
to  support  and  justify  the  PBL  strategy;  and  (5)  a  recommendation  and  summary 
of  the  implementation  plan  for  proceeding  with  the  best  value  alternative. 

The  Product  Support  Guide  further  states  that  a  BCA  should  be  updated  at 
appropriate  decision  points  when  sufficient  cost  and  perfonnance  data  are 
available  for  validating  the  assumptions  used,  including  the  costs  of  alternative 
approaches,  projected  cost  savings,  and  expected  performance  levels.  The 

Product  Support  Guide  also  concludes  that  the  BCA  must  stand  on  its  own 
and  be  able  to  withstand  the  rigorous  analysis  and  review  of  any  independent 
audit  agency. 

The  Acting  USD(AT&L)  issued  policy  memorandum  “Performance  Based 
Logistics  (PBL)  and  the  Business  Case  Analysis  (BCA),”  on  March  20,  2004. 

The  memorandum  states  that  the  Strategic  Planning  Guidance  requires  that  the 
Services  complete  a  BCA  by  September  30,  2006,  on  all  new  and  fielded 
Acquisition  Category  I  and  II  programs  for  application  of  PBL  sustainment 
strategies.  Further,  the  memorandum  states  that  the  assessment  should  be 
conducted  through  a  two-step  process,  with  Step  2  the  development  of  a  fonnal 
BCA. 

Although  the  TSSR  contract  predates  formal  PBL  BCA  requirements,  DoD 
guidance  at  that  time  required  that  DoD  evaluate  alternative  support  strategies. 
DoD  Instruction  7041.3,  “Economic  Analysis  for  Decisionmaking,”  November  7, 
1995,  requires  that  alternatives  be  evaluated  and  that  the  cost  and  benefits 
associated  with  each  alternative  under  consideration  be  quantified  and  ranked 
when  possible.  Our  review  of  the  TSSR  contract  files  showed  that,  although 
alternative  sustainment  support  strategies  were  under  consideration,  senior  Air 
Force  acquisition  officials  determined  that  the  TSSR  sustainment  approach  would 
be  chosen  and  a  cost  benefit  analysis  was  not  necessary. 

When  asked  whether  a  BCA  was  developed  for  the  Joint  STARS  weapon  system 
based  on  PBL  BCA  guidance  at  the  time,  the  SPM  provided  a  series  of  briefing 
charts  that  included  several  charts  described  as  the  BCA.  However,  the  SPM 
could  not  provide  support  for  any  of  the  projected  savings  for  supply  chain 
management,  depot  maintenance,  or  support  response  contained  in  the  charts  and 
did  not  include  an  analysis  of  alternatives. 

Because  the  TSSR  contract  is  more  than  5  years  old,  sufficient  cost  data  are 
available  for  the  SPM  to  reassess  the  support  strategy.  To  ensure  that  the  TSSR 
contract  is  the  optimum  support  strategy  and  meeting  warfighter  performance  and 
cost  requirements,  increasing  system  availability,  and  reducing  costs,  the  Joint 
STARS  SPM  should  answer  the  USD(AT&L)  requirement  and  complete  a  BCA 
for  the  Joint  STARS  weapon  system  by  September  30,  2006. 
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Developing  Performance  Outcomes 


While  the  TSSR  contract  contains  several  performance  measures,  it  does  not 
contain  performance  metrics  that  promote  the  PBL  goal  of  increasing  system 
readiness,  reducing  total  ownership  costs,  and  reducing  the  logistics  footprint. 

The  Product  Support  Guide  states  that  at  the  top  level,  perfonnance  outcomes  and 
corresponding  metrics  should  focus  on  warfighter  needs:  a  system  that  is 
operationally  available,  reliable,  and  effective,  with  minimal  logistics  footprint 
and  a  reasonable  cost. 

An  August  16,  2004,  memorandum  from  the  Acting  USD(AT&L),  “Performance 
Based  Logistics:  Purchasing  Using  Performance  Based  Criteria,”  states  that  for 
PBL,  performance  is  defined  in  tenns  of  military  objectives  using  operational 
availability,  operational  reliability,  cost  per  unit  usage,  the  logistics  footprint,  and 
logistics  response  time.  PBL  metrics  should  support  that  criteria.  In  addition,  the 
memorandum  provides  guidance  that  states  PBL  contracts  should  be  fixed  price. 
In  another  USD(AT&L)  memorandum  dated  November  22,  2005,  the  criteria 
from  the  August  16  memorandum  was  reiterated  and  directed  the  criteria’s  use  as 
the  standard  set  of  metrics  for  evaluating  overall  total  life-cycle  systems 
management. 

For  FY  2003  and  FY  2004,  Joint  STARS  was  below  the  warfighter’s  mission 
capability  requirement.  According  to  the  SSM,  factors  impacting  the  ability  of 
the  warfighter  to  meet  mission  capability  rates  were  engines,  fuels,  airframe,  and 
flight  control.  The  components,  referred  to  as  “bad  actors,”  are  organically 
maintained  through  Service-level  agreements  between  NGC  and  Air  Force 
depots.  DoD  and  Air  Force  guidance  both  include  increased  mission  capability 
as  a  PBL  goal.  Furthermore,  NGC  did  not  have  the  responsibility  for  the  “bad 
actors”  and  therefore,  did  not  have  total  control  over  Joint  STARS  mission 
capability  under  the  TSSR  contract. 

In  October  2004,  an  Air  Force  Logistics  Management  Agency  report  entitled, 
“Performance  Based  Logistics  Partnerships:  Assessment  of  Implementation 
Methodologies  for  Selected  Acquisition  Category  1  and  2  Systems,”  stated  that 
Air  Force  systems,  including  Joint  STARS,  were  not  using  high-level 
performance  metrics.  The  report  cited  mission  capability  rate,  improved  product 
affordability,  system  reliability,  and  logistics  footprint  as  examples  of  high-level 
metrics  that  measure  success  in  meeting  PBL  performance  goals. 

Officials  from  the  Air  Combat  Command  and  Air  Force  National  Guard  stated 
that  the  Air  Force  is  studying  mission  capability  rates  as  a  performance  driver. 
The  Air  Force  is  also  looking  into  ways  that  will  incorporate  the  Joint  STARS 
mission  capability  rate  into  the  TSSR  contract. 
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Management  Comments  on  the  Finding  and  Audit  Response 


The  Military  Deputy,  Office  of  the  Assistant  Secretary  of  the  Air  Force  (Acquisition) 
provided  comments  on  behalf  of  the  Assistant  Secretary  of  the  Air  Force  (Acquisition). 
The  Military  Deputy  partially  concurred  with  our  draft  finding  that  the  SPM  did  not  fully 
implement  PBL  initiatives  for  Joint  STARS,  stating  that  the  Joint  STARS  SPM  did  not 
retroactively  apply  PBL  principles  because  the  TSSR  contract  was  a  model  for 
sustainment  approach  before  PBL  guidance.  He  also  stated  that  the  SPM  agreed  to 
rebaseline  the  sustainment  cost  using  the  TSSR  contract  data  collected  over  the  last 
5  years,  which  will  form  the  basis  for  a  BCA  for  Joint  STARS  sustainment.  The  Military 
Deputy  provided  other  comments  on  the  draft  finding  which  mirror  their  comments  on 
each  recommendation.  Therefore,  we  discuss  each  of  these  in  the  Recommendation, 
Management  Comment,  and  Audit  Response  section  of  the  report. 


Recommendations,  Management  Comments,  and 
Audit  Response 


1.  We  recommend  that  the  Assistant  Secretary  of  the  Air  Force  (Acquisition) 
ensure  that  DoD  policy  memorandums  and  guides  are  distributed  to  each 
Product  Center  for  use  in  implementing  performance-based  logistics. 

Management  Comments.  In  responding  for  the  Assistant  Secretary  of  the  Air 
Force  (Acquisition),  the  Military  Deputy  concurred,  stating  that  the  Air  Force  will 
validate  that  policy  memorandums  and  guidance  for  PBL  are  available  to  each 
Product  Center,  Program  Executive  Offices,  and  SPMs. 

Audit  Response.  The  Air  Force  comments  are  responsive,  and  no  further 
comments  are  required. 

2.  We  recommend  that  the  System  Program  Manager  for  the  Joint 
Surveillance  Target  Attack  Radar  System: 

a.  Develop  performance-based  agreements  in  conjunction  with  the 
116th  Air  Control  Wing  based  on  warfighter  performance  requirements. 

b.  Develop  a  strategy  that  implements  into  the  total  system  support 
responsibility  contract  performance  requirements  obtained  from  the 
performance-based  agreements. 

Management  Comments.  In  responding  for  the  Assistant  Secretary  of  the  Air 
Force  (Acquisition),  the  Military  Deputy  partially  concurred  with 
Recommendations  2. a.  and  2.b.,  stating  that  the  Joint  STARS  program  office 
memorandum  of  agreement  specifies  the  level  of  support  for  warfighter 
requirements.  Further,  the  Task  Description  Document,  an  attachment  to  the 
memorandum  of  agreement,  integrates  the  negotiated  support  required  from  the 
contractor  to  manage  system  impacts.  The  Military  Deputy  also  stated  that  plans 
were  underway  to  revise  the  BCA  in  conjunction  with  the  planned  improvements 
to  the  scope,  objectivity,  and  performance  of  their  performance-based  agreements. 
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The  effort  will  provide  an  avenue  for  the  SPM  to  improve  warfighter  support  and 
overall  weapon  system  performance  for  Joint  STARS. 


Audit  Response.  Although  the  Military  Deputy  partially  concurred,  the 
memorandum  of  agreement  was  not  clear  in  specifying  warfighter  requirements. 
Therefore,  the  Task  Description  Document  did  not  clearly  specify  the  warfighter 
requirements  for  the  Defense  contractor.  However,  the  Military  Deputy’s  stated 
actions  to  revise  the  BCA  in  conjunction  with  the  planned  improvements  to  the 
scope,  objectivity,  and  performance  of  their  PBL  agreements  are  responsive.  No 
further  comments  are  required. 

c.  Analyze  current  Joint  STARS  cost  and  resource  data  and  develop 
baselines  for  use  in  developing  a  business  case  analysis. 

Management  Comments.  In  responding  for  the  Assistant  Secretary  of  the  Air 
Force  (Acquisition),  the  Military  Deputy  concurred  and  stated  that  the  planned 
revision  to  the  BCA  will  encompass  accumulated  contract  performance  data  over 
the  last  5  years  in  rebaselining  the  cost  and  performance  history,  which  will  shape 
the  scope  and  content  of  future  options  for  Joint  STARS.  The  expected 
completion  date  for  the  action  is  September  30,  2006. 

Audit  Response.  The  Air  Force  comments  are  responsive,  and  no  further 
comments  are  required. 

d.  Develop  a  business  case  analysis  using  baselines  developed  in 
Recommendations  2.a.  and  2.c.  above  that  would  meet  the  deadline  of 
September  30,  2006,  that  the  Under  Secretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics  established.  The  business  case  analysis  should 
include  an  evaluation  of  alternative  sustainment  support  strategies  and  cost 
benefits  for  the  Joint  STARS. 

Management  Comments.  In  responding  for  the  Assistant  Secretary  of  the  Air 
Force  (Acquisition),  the  Military  Deputy  partially  concurred,  stating  that  even 
though  the  Joint  STARS  program  meets  the  intent  of  the  USD(AT&L) 
implementation  guidance,  the  Air  Force  will  develop  a  forward-looking  BCA. 

The  Military  Deputy  further  stated  that  he  will  assess  alternative  solutions  that 
meet  perfonnance  objectives  of  the  warfighter  and  conduct  an  analysis  that  will 
consider  quantifiable  and  nonquantifiable  factors  that  support  an  optimum 
investment  decision.  The  expected  completion  date  for  the  action  is  March  3 1 , 
2007. 

Audit  Response.  Although  the  Military  Deputy  partially  concurred,  the  BCA  it 
had  in  place  did  not  meet  the  intent  of  the  guidance  that  the  USD(AT&L)  issued. 
However,  the  Military  Deputy’s  agreement  to  develop  a  BCA  is  responsive. 
Although  the  expected  date  of  completion  does  not  meet  the  USD(AT&L) 
required  date,  we  accept  the  Air  Force’s  actions  to  develop  a  forward-looking 
BCA  in  accordance  with  the  policy  guidance.  No  further  comments  are  required. 
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e.  Develop  performance  measures  for  the  Joint  STARS  total  system 
support  responsibility  contract  that  will  promote  the  goal  of  increasing 
system  readiness,  reducing  total  life-cycle  costs,  and  reducing  the  logistics 
footprint. 

Management  Comments.  In  responding  for  the  Assistant  Secretary  of  the  Air 
Force  (Acquisition),  the  Military  Deputy  partially  concurred  and  stated  that 
performance  measures  that  promote  the  goal  of  system  readiness,  reduction  in 
total  life-cycle  costs,  and  reduction  of  logistics  footprint  were  already  in  the 
memorandum  of  agreement.  Further,  the  Military  Deputy  stated  that  the  Air 
Force  uses  a  continuous  improvement  process  that  is  based  on  warfighter 
requirements  and  experience.  The  impending  revision  to  the  BCA  in  conjunction 
with  PBAs  will  provide  avenues  for  the  SPM  to  improve  warfighter  support  and 
overall  weapon  system  performance. 

Audit  Response.  The  Military  Deputy’s  comments  were  partially  responsive. 

The  memorandum  of  agreement  was  not  clear  that  they  had  in  place  goals  of 
increasing  system  readiness,  reduction  in  total  life-cycle  costs,  and  reductions  in 
logistics  footprint.  Although  their  efforts  to  revise  the  BCA  in  conjunction  with 
PBAs  will  assist  in  promoting  those  goals,  we  request  that  the  Assistant  Secretary 
of  the  Air  Force  (Acquisition)  provide  clarification  on  his  actions  to  develop  new 
performance  measures  in  meeting  these  goals  and  provide  additional  comments  in 
response  to  the  final  report. 

f.  Analyze  the  impact  of  transitioning  the  total  system  support 
responsibility  contract  from  a  cost-plus  contract  to  a  firm-fixed 
priced  contract. 

Management  Comments.  In  responding  for  the  Assistant  Secretary  of  the  Air 
Force  (Acquisition),  the  Military  Deputy  concurred  and  stated  the  expected 
completion  date  for  this  action  is  March  31,  2007. 

Audit  Response.  The  Air  Force  comments  are  responsive,  and  no  further 
comments  are  required. 
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Appendix  A.  Scope  and  Methodology 


We  performed  the  audit  at  the  Joint  STARS  Joint  Program  Office  located  at  the 
Electronic  Systems  Center,  Hanscom  Air  Force  Base,  Massachusetts  and  at 
Warner  Robins  Air  Logistics  Center,  Robins  Air  Force  Base,  Georgia.  We  also 
conducted  audit  work  at  the  Defense  Contract  Management  Agency  in 
Melbourne,  Florida.  We  contacted  personnel  at  the  Office  of  the  USD(AT&L); 
the  office  of  the  Assistant  Secretary  of  the  Air  Force  (Acquisition);  the  office  of 
the  Assistant  Secretary  of  the  Air  Force  (Installation  and  Logistics);  the  Air 
Combat  Command;  the  Air  National  Guard;  and  the  Defense  Logistics  Agency. 
We  reviewed  documents  dated  from  November  1995  to  November  2005. 

We  assessed  the  adequacy  of  the  Joint  STARS  program  office  implementation  of 
PBL  by  reviewing  DoD  and  Air  Force  policies  and  regulations  regarding 
responsibilities  and  procedures  for  implementing  PBL.  We  also  reviewed  the 
Quadrennial  Defense  Review  and  DoD  directives  pertaining  to  PBL.  We 
reviewed  the  adequacy  of  the  BCA  by  reviewing  DoD  and  Air  Force  guidance. 
We  assessed  the  TSSR  contract  for  PBL  provisions. 

We  interviewed  program  office  officials  pertaining  to  the  various  aspects  of 
implementing  PBL  and  incorporating  PBL  requirements  in  the  TSSR  contract. 
Discussions  included  the  sustainment  approach  for  Joint  STARS,  development  of 
PBAs,  baselines,  and  performance  outcomes  as  well  as  the  use  of  both  fixed-price 
and  cost  plus  fixed-fee  contracts  to  conduct  scheduled  depot  maintenance. 

Further,  we  interviewed  officials  at  Defense  Contract  Management  Agency  to 
inquire  about  their  monitoring  of  the  TSSR  contract  and  their  reporting  to  the 
program  office. 

We  perfonned  this  audit  from  October  2004  through  February  2006  in  accordance 
with  generally  accepted  government  auditing  standards. 

Use  of  Computer-Processed  Data.  We  did  not  use  computer-processed  data  to 
perform  this  audit. 

Use  of  Technical  Assistance.  We  did  not  rely  on  the  assistance  from  the 
members  of  the  Quantitative  Methods  Division  and  the  Technical  Assessment 
Division  of  the  OAIG-AUD. 

Government  Accountability  Office  High-Risk  Area.  The  Government 
Accountability  Office  (GAO)  has  identified  several  high-risk  areas  in  DoD.  This 
report  provides  coverage  of  the  Defense  Weapon  Systems  Acquisition  high-risk 
area. 
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Appendix  B.  Prior  Coverage 


During  the  last  5  years,  GAO,  the  DoD  Inspector  General  (IG),  and  the  Navy 
issued  14  reports  related  to  implementing  PBL.  Unrestricted  GAO  reports  can  be 
accessed  over  the  Internet  at  http://www.gao.gov.  Unrestricted  DoD  IG  reports 
can  be  accessed  at  http://www.dodig.mil/audit/reports. 


GAO 


GAO  Report  No.  GAO-05-966,  “Defense  Management:  DoD  Needs  to 
Demonstrate  That  Performance-Based  Logistics  Contract  Are  Achieving 
Expected  Benefits,”  September  9,  2005 

GAO  Report  No.  GAO-04-715,  “Defense  Management:  Opportunities  to 
Enhance  the  Implementation  of  Performance-Based  Logistics,”  August  16,  2004 

GAO  Report  No.  GAO-02- 1049,  “Contract  Management:  Guidance  Needed  for 
Using  Performance-Based  Service  Contracting,”  September  23,  2002 

GAO  Report  No.  GAO-02-306,  “Defense  Logistics:  Opportunities  to  Improve 
the  Army’s  and  the  Navy’s  Decision-making  Process  for  Weapons  Systems 
Support,” 

February  28,  2002 

GAO  Report  No.  GAO-0 1-6 18,  “Defense  Logistics:  Air  Force  Lacks  Data  to 
Assess  Contractor  Logistics  Support  Approaches,”  September  7,  2001 


DoD  IG 


DoD  IG  Report  No.  D-2005-037,  “Implementation  of  Performance  Based 
Logistics  for  the  Javelin  Weapon  System,”  March  7,  2005 

DoD  IG  Report  No.  D-2004-1 10,  “The  Military  Departments’  Implementation  of 
Performance-Based  Logistics  in  Support  of  Weapon  Systems,”  August  23,  2004 

DoD  IG  Report  No.  D-2004-021,  “Effectiveness  of  Maintenance  Work  Performed 
Under  Contract  FA4452-01-C-0001  at  Andrews  Air  Force  Base,” 

November  19,  2003 

DoD  IG  Report  No.  D-2003-120,  “F/A-18E/F  Integrated  Readiness  Support 
Teaming  Program,”  August  8,  2003 

DoD  IG  Report  No.  D-2002-1 12,  “Industrial  Prime  Vendor  Program  at  the  Air 
Force  Air  Logistics  Centers,”  June  20,  2002 
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DoD  IG  Report  No.  D-2000-180,  “Commercial  Contract  for  Total  Logistics 
Support  of  Aircraft  Auxiliary  Power  Units,”  August  31,  2000 


Navy 


Naval  Audit  Service  Report  No.  N2003-0050,  “Contractor  Logistics  Support 
Oversight,”  May  15,  2003 

Naval  Audit  Service  Report  No.  N2003-0024,  “Contractor  Logistics  Support  at 
the  Naval  Air  Systems  Command,”  January  29,  2003 

Naval  Audit  Service  Report  No.  N2002-0069,  “Contractor  Logistics  Support  at 
the  Space  and  Naval  Warfare  Systems  Command,”  August  8,  2002 


14 


Appendix  C.  Report  Distribution 


Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  (Acquisition 
Technology  and  Logistics) 

Under  Secretary  of  Defense  (Comp troller)/Chief  Financial  Officer 
Deputy  Chief  Financial  Officer 
Deputy  Comptroller  (Program/Budget) 

Director,  Program  Analysis  and  Evaluation 
Director,  Defense  Procurement  and  Acquisition  Policy 


Department  of  the  Army 

Assistant  Secretary  of  the  Army  (Financial  Management  and  Comptroller) 
Auditor  General,  Department  of  the  Anny 


Department  of  the  Navy 

Naval  Inspector  General 

Auditor  General,  Department  of  the  Navy 


Department  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  (Financial  Management  and  Comptroller) 
Assistant  Secretary  of  the  Air  Force  (Acquisition) 

Auditor  General,  Department  of  the  Air  Force 
Director,  Air  Combat  Command 
Director,  Air  National  Guard 

Commander,  Program  Executive  Office,  Electronic  Systems  Center,  Hanscom  AFB 
System  Project  Manager,  Joint  Surveillance  Target  Attack  Radar  System,  Hanscom 
AFB 

System  Support  Manager,  Warner  Robins  Air  Logistics  Center 


Other  Defense  Organizations 

Director,  Defense  Contract  Audit  Agency 
Director,  Defense  Contract  Management  Agency 


Non-Defense  Federal  Organization 

Office  of  Management  and  Budget 
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Congressional  Committees  and  Subcommittees,  Chairman  and 
Ranking  Minority  Member 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 
Senate  Committee  on  Armed  Services 
Senate  Committee  on  Governmental  Affairs 
House  Committee  on  Appropriations 

House  Subcommittee  on  Defense,  Committee  on  Appropriations 
House  Committee  on  Anned  Services 
House  Committee  on  Government  Reform 

House  Subcommittee  on  Government  Management,  Finance,  and  Accountability, 
Committee  on  Government  Refonn 

House  Subcommittee  on  National  Security,  Emerging  Threats,  and  International 
Relations,  Committee  on  Government  Reform 
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Department  of  the  Air  Force  Comments 


MEMORANDUM  FOR  DEPARTMENT  OF  DEFENSE  INSPECTOR  GENERAL 

ATTN:  DEPUTY  INSPECTOR  GENERAL  FOR  AUDITING 

FROM:  SAF/AQ 

SUBJECT:  Draft  Audit  Report,  Dated  16  March  06,  Project  No.  D2005-D000LH-0046.00 

Implementation  of  Performance-Based  Logistics  (PBL)  for  the  Joint  Surveillance 
Target  Attack  Radar  System  (Joint  STARS) 

Thank  you  for  the  time  and  effort  your  team  invested  in  examining  the  Joint  STARS 
implementation  of  PBL.  I  also  appreciate  the  opportunity  for  the  Air  Force  to  review  the  draft 
audit  report.  Air  Force  comments  are  summarized  below: 

a.  Finding  1  -  The  Air  Force  partially  concurs  with  the  finding  that  the  Joint  STARS 
System  Program  Manager  (SPM)  did  not  fully  implement  PBL  initiatives  for  the  Joint 
STARS  weapon  system.  The  Joint  Stars  SPM  did  not  retroactively  apply  PBL 
principles  given  the  Total  System  Support  Responsibility  (TSSR)  contract,  which  was 
in  place  before  the  PBL  audit,  and  was  an  accepted  and  model  sustainment  approach 
prior  to  official  PBL  guidance.  In  accordance  with  Air  Force  and  OSD  policy 
guidance,  the  SPM  has  agreed  to  re-baseline  the  program  sustainment  cost  using  the 
TSSR  contract  data  collected  over  the  past  five  years.  This  will  also  form  the  basis 
for  the  business  case  analysis  (BCA)  of  PBL  approaches  for  use  in  future  sustainment 
strategy  development. 

b.  Finding  2  -  The  Air  Force  partially  concurs  with  the  finding  that  the  Joint  STARS 
sustainment  strategy  lacks  support  of  fully  developed  Performance  Based  Agreements 
(PBA)  with  the  warfighter.  The  current  Memorandum  Of  Agreement  (MOA)  meets 
the  minimum  intent  of  a  PBA.  The  MOA  specifies  the  level  of  product  support  and 
warfighter  requirements.  The  Task  Description  Document,  w'hich  is  an  attachment  to 
the  MOA,  integrates  the  negotiated  support  required  from  Northrop  Grumman 
Corporation  to  manage  system  impacts  throughout  the  life  cycle;  development, 
production,  and  sustainment  w  ith  the  successful  achievement  of  the  warfighter’s 
product  support  requirements.  The  impending  revision  to  the  BCA  in  conjunction 
with  the  planned  improvements  to  the  scope,  objectivity  and  performance  of  their 
PBL  agreements  will  continue  to  provide  avenues  for  the  SPM  to  improve  warfighter 
support  and  overall  w'eapon  system  performance. 


c.  Finding  3  -  The  Air  Force  partially  concurs  with  the  finding  that  the  Joint  STARS 
SPM  did  not  develop  cost  and  performance  baselines  for  the  Joint  STARS  weapon 


system  before  the  August  2000  approval  by  the  Air  Force  acquisition  officials  for  the 
overall  Joint  STARS  sustainment  strategy.  Limited  fleet  size  impacted  the  SPM’s 
ability  to  ensure  that  an  adequate  cost  and  performance  baseline  was  accomplished 
in  August  2000  prior  to  the  TSSR  contract  award.  Now  that  all  17  aircraft  have  been 
delivered,  and  a  statistically  significant  number  of  flying  hours  logged,  the  SPM  plans 
to  develop  an  updated  cost  and  performance  baseline  from  the  past  five  years  of 
accumulated  data. 

d.  Finding  4  -  The  Air  Force  partially  concurs  with  the  finding  the  Joint  STARS  SPM 
did  not  prepare  a  BCA  to  provide  analysis  of  potential  alternative  support  strategies. 
The  BCA,  prepared  in  May  2000,  predates  formal  PBL  BCA  requirements.  Utilizing 
the  TSSR  contract  data  gathered  over  the  past  five  years,  the  SPM  is  preparing  to 
revise  the  BCA  and  PBL  approach  to  ensure  optimization  of  the  support  strategy  in 
meeting  all  of  the  warfighters  performance  and  cost  requirements. 

e.  Finding  5  -  The  Air  Force  partially  concurs  with  the  finding  that  while  the  TSSR 
contract  contains  several  performance  measures,  that  it  does  not  contain  performance 
metrics  that  promote  the  PBL  goal  of  increasing  system  readiness,  reducing  total 
ownership  costs,  and  reduction  in  the  logistics  foot  print.  The  TSSR  contract  has 
several  parameters  that  reflect  a  close  collaborative  relationship  between  the  Program 
Office  and  the  warfighter  that  demonstrates  an  effective  support  of  improved  system 
readiness,  system  cost  reduction  and  reduction  in  logistics  footprint.  The  impending 
revision  to  the  BCA  in  conjunction  with  the  planned  improvements  to  the  scope, 
objectivity  and  performance  of  their  PBL  agreements  will  continue  to  provide 
avenues  for  the  SPM  to  improve  warfighter  support  and  overall  weapon  system 
performance. 

f.  Recommendation  1 .  -  The  Air  Force  concurs  with  the  recommendation  to  ensure  that 
pertinent  DoD  policy  memorandums  and  PBL  guidance  is  made  available  to  each 
product  center  for  use  in  implementing  PBL.  We  will  validate  that  pertinent  guidance 
documents  are  available  to  all  Program  Executive  Offices  and  SPMs. 

g.  Recommendation  2a.  -  The  Air  Force  partially  concurs  with  the  recommendation  that 
the  Joint  STARS  program  develop  PBAs  in  conjunction  with  the  1 16'*1  Air  Control 
Wing  based  on  warfighter  requirements.  The  MOA  specifies  the  level  of  product 
support  and  warfighter  requirements.  The  Task  Description  Document,  which  is  an 
attachment  to  the  MOA,  integrates  the  negotiated  support  required  from  Northrop 
Grumman  Corporation  to  manage  system  impacts  throughout  the  life  cycle: 
development,  production,  and  sustainment  with  the  successful  achievement  of  the 
warfighter’s  product  support  requirements.  The  impending  revision  to  the  BCA  in 
conjunction  with  the  planned  improvements  to  the  scope,  objectivity  and  performance 
of  their  PBL  agreements  w  ill  continue  to  provide  avenues  for  the  SPM  to  improve 
warfighter  support  and  overall  weapon  system  performance. 

h.  Recommendation  2b.  —  The  Air  Force  partially  concurs  with  recommendation  to 
develop  a  strategy  that  implements  into  the  TSSR  contract  performance  requirements 
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obtained  from  the  PBAs.  The  current  MO  As  do  account  for  the  warfighter 
performance  requirements.  The  Task  Description  Document,  which  is  an 
attachment  to  the  MO  A,  integrates  the  negotiated  support  required  from  Northrop 
Grumman  Corporation  to  manage  system  impacts  throughout  the  life  cycle: 
development,  production,  and  sustainment  with  the  successful  achievement  of  the 
warfighter’s  product  support  requirements.  However,  the  impending  revision  to  the 
BCA  in  conjunction  with  the  planned  improvements  to  the  scope,  objectivity  and 
performance  of  their  PBL  agreements  will  continue  to  provide  avenues  for  the  SPM 
to  improve  warfighter  support  and  overall  weapon  system  performance. 

i.  Recommendation  2c.  -  The  Air  Force  concurs  with  the  recommendation  to  establish 
new  baselines  to  reflect  the  extensive  data  and  experience  gained  during  the  first  five 
years  of  the  TSSR  contract.  The  scheduled  revision  to  the  current  BCA  will  use  the 
TSSR  data  accumulated  over  the  past  five  years  to  re-baseline  the  cost  and 
performance  of  the  Joint  STARS  weapon  system  and  confirm  the  efficacy  of  the 
current  TSSR  contract  to  shape  the  scope  and  content  of  future  options. 

Estimated  Completion  Date  (ECD):  30  Sep  06. 

j.  Recommendation  2d.  -  The  Air  Force  partially  concurs  with  the  recommendation  to 
develop  a  BCA  using  baselines  developed  from  recommendations  2a  and  2c.  Even 
though  the  Joint  STARS  program  meets  the  intent  of  the  USD  (AT&L) 
implementation  guidance,  a  forward-looking  BCA  will  be  conducted  in  accordance 
with  the  policy  guidance.  This  BCA  will  assess  and  determine  possible  alternative 
solutions  in  terms  of  their  ability  to  meet  the  logistics  performance  objectives  of  the 
warfighter.  It  will  be  a  best-value  analysis  that  will  consider  not  only  cost  but  also 
other  quantifiable  and  non-quantifiable  factors  supporting  an  optimum  investment 
decision.  To  avoid  any  impact  to  the  TSSR  contract,  the  SPM’s  ECD  for  this  action 
is  3 1  Mar  07. 

k.  Recommendation  2e.  -  The  Air  Force  partially  concurs  with  the  recommendation  to 
develop  performance  measures  for  the  Joint  STARS  TSSR  contract  promoting  the 
goal  of  increasing  system  readiness,  reduction  in  total  life-cycle  costs,  and  reduction 
of  logistics  foot  print  given  they  are  already  in  place.  The  Air  Force  uses  a 
continuous  improvement  process  based  on  warfighter  requirements  and  operational 
experience.  The  MOA  specifies  the  level  of  product  support  and  warfighter 
requirements.  The  Task  Description  Document,  which  is  an  attachment  to  the  MOA, 
integrates  the  negotiated  support  required  from  Northrop  Grumman  Corporation  to 
manage  system  impacts  throughout  the  life  cycle:  development,  production,  and 
sustainment  with  the  successful  achievement  of  the  warfighter’s  product  support 
requirements.  The  impending  revision  to  the  BCA  in  conjunction  with  the  planned 
improvements  to  the  scope,  objectivity  and  performance  of  their  PBL  agreements  will 
continue  to  provide  avenues  for  the  SPM  to  improve  warfighter  support  and  overall 
weapon  system  performance. 
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1.  Recommendation  2f.  -  The  Air  Force  concurs  with  the  recommendation  to  analyze 
the  impact  of  transitioning  the  TSSR  contract  from  a  cost-plus  to  a  firm  fixed  priced 
contract.  To  avoid  impact  on  the  current  TSSR  contract,  the  SPMs  ECD  for  this 
action  is  3 1  Mar  07. 

The  SAF/AQ  staff  stands  ready  to  assist  in  any  way  to  ensure  all  concerns  are  addressed 
prior  to  the  final  report  publication.  My  point  of  contact  is  Mr.  Mark  Humphrey, 
mark.humnhrev@pentagon.af.mil.  703-588-7105. 

DONALD  J.  i/OFFMAN,  Lt  Gen,  USAF 
Military  Deputy,  Office  of  the  Assistant  Secretary' 
of  the  Air  Force  (Acquisition) 
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